Method and system for network configuration discovery in a network manager

ABSTRACT

A method and system for managing an interconnect fabric that connects nodes. A network manager manages an interconnect fabric or network of routing devices (e.g., interconnect fabric modules, switches, or routers) to allow source nodes to transmit data to destination nodes. The network manager receives registration requests from source nodes to send data to destination nodes, configures the routing devices of the network to establish a path from each source node to its destination node, and provides a virtual address to each source node. The virtual address identifies a path from the source node to the destination node. The source node sends the data to its destination node by providing the data along with the virtual address to a routing device of the network. Upon receiving the data and the virtual address, a source-side port of each routing device in the path uses the virtual address to identify a destination-side port through which the data and the virtual address are to be transmitted. The network manager configures the routing devices by setting the mappings from a source-side port to a destination-side port for each routing device in the path.

CROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application claims the benefit of U.S. ProvisionalApplication No. 60/287,069 entitled “METHOD FOR IMPLEMENTING A CLUSTERNETWORK FOR HIGH PERFORMANCE AND HIGH AVAILABILITY USING A FIBRE CHANNELSWITCH FABRIC,” filed Apr. 27, 2001; U.S. Provisional Application No.60/287,120 entitled “MULTI-PROTOCOL NETWORK FOR ENTERPRISE DATACENTERS,” filed Apr. 27, 2001; U.S. Provisional Application No.60/286,918 entitled “UNIFIED ENTERPRISE NETWORK SWITCH (UNEX) PRODUCTSPECIFICATION,” filed Apr. 27, 2001; U.S. Provisional Application No.60/286,922 entitled “QUALITY OF SERVICE EXAMPLE,” filed Apr. 27, 2001;U.S. Provisional Application No. 60/287,081 entitled “COMMUNICATIONSMODEL,” filed Apr. 27, 2001; U.S. Provisional Application No. 60/287,075entitled “UNIFORM ENTERPRISE NETWORK SYSTEM,” filed Apr. 27, 2001; U.S.Provisional Application No. 60/314,088 entitled “INTERCONNECT FABRICMODULE,” filed Aug. 21, 2001; U.S. Provisional Application No.60/314,287 entitled “INTEGRATED ANALYSIS OF INCOMING DATATRANSMISSIONS,” filed Aug. 22, 2001; U.S. Provisional Application No.60/314,158 entitled “USING VIRTUAL IDENTIFIERS TO ROUTE TRANSMITTED DATATHROUGH A NETWORK,” filed Aug. 21, 2001, and is related to U.S. patentapplication Ser. No. ______ entitled “METHOD AND SYSTEM FOR VIRTUALADDRESSING IN A COMMUNICATIONS NETWORK,” (Attorney Docket No. 030048019US1); U.S. patent application Ser. No. ______ entitled “METHOD ANDSYSTEM FOR LABEL TABLE CACHING IN A ROUTING DEVICE,” (Attorney DocketNo. 030048024US); U.S. patent application Ser. No. ______ entitled“METHOD AND SYSTEM FOR MULTIFRAME BUFFERING IN A ROUTING DEVICE,”(Attorney Docket No. 030048025US); U.S. patent application Ser. No.______ entitled “METHOD AND SYSTEM FOR DOMAIN ADDRESSING IN ACOMMUNICATIONS NETWORK,” (Attorney Docket No. 030048026US); U.S. patentapplication Ser. No. ______ entitled “METHOD AND SYSTEM FOR INTERSWITCHLOAD BALANCING IN A COMMUNICATIONS NETWORK,” (Attorney Docket No.030048027US); U.S. patent application Ser. No. ______ entitled “METHODAND SYSTEM FOR INTERSWITCH DEADLOCK AVOIDANCE IN A COMMUNICATIONSNETWORK,” (Attorney Docket No. 030048028US); U.S. patent applicationSer. No. ______ entitled “METHOD AND SYSTEM FOR CONNECTION PREEMPTION INA COMMUNICATIONS NETWORK,” (Attorney Docket No. 030048029US); U.S.Patent application Ser. No. ______ entitled “METHOD AND SYSTEM FORMULTICASTING IN A ROUTING DEVICE,” (Attorney Docket No. 030048030US);U.S. patent application Ser. No. ______ entitled “METHOD AND SYSTEM FORNETWORK CONFIGURATION DISCOVERY IN A NETWORK MANAGER,” (Attorney DocketNo. 030048032US); U.S. patent application Ser. No. ______ entitled“METHOD AND SYSTEM FOR PATH BUILDING IN A COMMUNICATIONS NETWORK,”(Attorney Docket No. 030048033US); U.S. patent application Ser. No.______ entitled “METHOD AND SYSTEM FOR RESERVED ADDRESSING IN ACOMMUNICATIONS NETWORK,” (Attorney Docket No. 030048035US); U.S. patentapplication Ser. No. ______ entitled “METHOD AND SYSTEM FORRECONFIGURING A PATH IN A COMMUNICATIONS NETWORK,” (Attorney Docket No.030048036US1); U.S. patent application Ser. No. ______ entitled “METHODAND SYSTEM FOR ADMINISTRATIVE PORTS IN A ROUTING DEVICE,” (AttorneyDocket No. 030048037US); U.S. patent application Ser. No. ______entitled “PARALLEL ANALYSIS OF INCOMING DATA TRANSMISSIONS,” (AttorneyDocket No. 030048038US); U.S. patent application Ser. No. ______entitled “INTEGRATED ANALYSIS OF INCOMING DATA TRANSMISSIONS,” (AttorneyDocket No. 030048039US); U.S. patent application Ser. No. ______entitled “USING VIRTUAL IDENTIFIERS TO ROUTE TRANSMITTED DATA THROUGH ANETWORK,” (Attorney Docket No. 030048040US); U.S. patent applicationSer. No. ______ entitled “USING VIRTUAL IDENTIFIERS TO PROCESS RECEIVEDDATA ROUTED THROUGH A NETWORK,” (Attorney Docket No. 030048041US); U.S.patent application Ser. No. ______ entitled “METHOD AND SYSTEM FORPERFORMING SECURITY VIA VIRTUAL ADDRESSING IN A COMMUNICATIONS NETWORK,”(Attorney Docket No. 030048042US); and U.S. Patent application Ser. No.______ entitled “METHOD AND SYSTEM FOR PERFORMING SECURITY VIADE-REGISTRATION IN A COMMUNICATIONS NETWORK” (Attorney Docket No.030048043US), which are all hereby incorporated by reference in theirentirety.

TECHNICAL FIELD

[0002] The described technology relates to a network manager for routingdevices of an interconnect fabric.

BACKGROUND

[0003] The Internet has emerged as a critical commerce andcommunications platform for businesses and consumers worldwide. Thedramatic growth in the number of Internet users, coupled with theincreased availability of powerful new tools and equipment that enablethe development, processing, and distribution of data across theInternet have led to a proliferation of Internet-based applications.These applications include e-commerce, e-mail, electronic filetransfers, and online interactive applications. As the number of usersof, and uses for, the Internet increases so does the complexity andvolume of Internet traffic. According to UUNet, Internet traffic doublesevery 100 days. Because of this traffic and its business potential, agrowing number of companies are building businesses around the Internetand developing mission-critical business applications to be provided bythe Internet.

[0004] Existing enterprise data networks (“EDNs”) that supporte-commerce applications providing services to customers are strainingunder the demand to provide added performance and added services. Thegrowing customer demands for services, along with a highly competitivemarket, has resulted in increasingly complex ad hoc EDNs. Affordable,high-performance EDN solutions require extensive scalability, very highavailability, and ease of management. These attributes are significantlycompromised or completely lost as existing solutions are grown to meetthe demand.

[0005] Current architectures of EDNs typically include threesub-networks: 1) a local area network (LAN) for web and databaseservers, 2) a computational network for application servers, and 3) astorage area network (SAN). The processing and storage elements attachedto these sub-networks may have access to a wide area network (WAN) ormetropolitan area network (MAN) through a bridging device commonly knownas an edge switch. Each of these sub-networks typically uses a distinctprotocol and associated set of hardware and software including networkinterface adapters, network switches, network operating systems, andmanagement applications. Communication through the EDN requires bridgingbetween the sub-networks that requires active participation of serverprocessing resources for protocol translation and interpretation.

[0006] There are many disadvantages to the current architecture of EDNs.The disadvantages result primarily because the multi-tiered architectureis fractured and complex. First, it is very difficult to integrate thedisparate systems that use different communications protocols,interfaces, and so on. Second, overall performance suffers because eachsub-network is managed separately, rather than being managed withcomprehensive knowledge of the complete network. Third, the cost ofmaintaining three disparate types of network hardware and software canbe high. Fourth, it is difficult to scale an architecture that uses suchdisparate systems. It would be desirable to have an architecture forEDNs that would be alleviate the many disadvantages of the currentfractured multi-tiered architectures.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007]FIG. 1 is a network diagram illustrating various nodes of anexample Fibre Channel fabric-based interconnect network that areinter-communicating using virtual identifiers.

[0008]FIG. 2 is a flow diagram illustrating the discovery processing ofa component of the interconnect fabric module in one embodiment.

[0009]FIG. 3 is a flow diagram illustrating the discovery processing ofthe network manager in one embodiment.

[0010]FIG. 4 is a flow diagram illustrating the process of establishinga path by the network manager in one embodiment.

[0011]FIG. 5 is a flow diagram illustrating the processing of anidentify virtual address component of the network manager in oneembodiment.

[0012]FIG. 6 is a flow diagram illustrating the processing of aninitialize label table component of the network manager in oneembodiment.

[0013]FIG. 7 is a block diagram illustrating a distributed networkmanager in one embodiment.

[0014]FIG. 8 is a flow diagram illustrating the processing of acomponent of an interconnect fabric module that processes reservedaddresses in one embodiment.

DETAILED DESCRIPTION

[0015] A method and system for managing an interconnect fabric thatconnects nodes is provided. In one embodiment, a network manager managesan interconnect fabric or network of routing devices (e.g., interconnectfabric modules, switches, or routers) to allow source nodes to transmitdata to destination nodes. The network manager receives registrationrequests from source nodes to send data to destination nodes, configuresthe routing devices of the network to establish a path from each sourcenode to its destination node, and provides a virtual address to eachsource node. The virtual address identifies a path from the source nodeto the destination node. The source node sends the data to itsdestination node by providing the data along with the virtual address toa routing device of the network. Upon receiving the data and the virtualaddress, a source-side port of each routing device in the path uses thevirtual address to identify a destination-side port through which thedata and the virtual address are to be transmitted. The network managerconfigures the routing devices by setting the mappings from asource-side port to a destination-side port for each routing device inthe path. The routing devices receive data via source-side ports andtransmits data via destination-side ports.

[0016] In one embodiment, the network manager may be centralized ordistributed. A centralized network manager may reside at one nodeconnected to the interconnect fabric. The centralized network managermay provide configuration information to the routing devices usingin-band communications or out-of-band communications. In-bandcommunications refers to the use of the communications links connectingthe ports of the routing devices. Out-of-band communications refers tothe use of communications links used specifically to connect the routingdevices to the network manager. A centralized network manager mayalternatively reside within a routing device. Each routing device mayhave the capabilities to be the network manager. Upon initialization,the routing devices may coordinate to select which of the routingdevices is to function as the network manager. A distributed networkmanager, in contrast, may have its functions performed at variousmanager devices connected directly to the routing devices. The networkmanager at each manager device can control the routing device(s) towhich it is directly connected. In addition, the network manager at eachmanager device can communicate with the network managers at othermanager devices via in-band or out-of-band communications to coordinatecontrol of the routing devices. In one embodiment, the distributednetwork manager can have different functions performed at variousmanager devices.

[0017] In one embodiment, the network manager identifies paths throughthe network from source nodes to destination nodes. The paths may beidentified initially when the network manager starts up, may beidentified when the network topology (e.g., the routing devices of thenetwork and their interconnections) changes (e.g., as a result of afailure), or may be identified dynamically when a registration requestis received from a source node. One skilled in the art will appreciatethat various combinations of these techniques for identifying a path maybe used. For example, the network manager may identify paths dynamicallyat registration, but may re-identify paths when the topology of thenetwork changes. Regardless of which of these techniques are used, thenetwork manager would typically need to know the topology of the networkto identify the paths.

[0018] In one embodiment, the network manager dynamically discovers thetopology of the network at initialization. The network manager candiscover the topology in several different ways. The network manager canbe provided with configuration information that identifies the routingdevices of the network. The network manager can use this configurationinformation to send a message to each routing device asking which of itsports are connected to another device. The network manager can then senda query message via each connected port asking the connected-to deviceto identify itself and its port. From the responses to the querymessages, the network manager can identify the connections (i.e.,communications links) between the routing devices and thus the topologyof the network. Alternatively, rather than sending a query message toeach connected-to port, the routing devices upon initialization canrequest the connected-to devices to provide their identifications. Therouting devices can then provide the identifications of the connected-toports to the network manager. The configuration information along withthe identifications of the connected-to ports describes the networktopology.

[0019] In another embodiment, the network manager can dynamicallydiscover the identifications of the routing devices by sending querymessages through the ports of the routing device to which it is directlyconnected. The network manager then becomes aware of each routing devicethat responds to the query. The network manager then sends a querymessage through the ports of each responding routing device.Alternatively, the network manager can send one query message to therouting device to which it is directly connected and that routing devicecan forward the query message via each of its ports to the routingdevice to which it is directly connected. Each port upon receiving thequery message may send a message to the network manager with itsidentification along with the identification of the port to which it isdirectly connected.

[0020] In one embodiment, each routing device may dynamically discoverwhich of its ports are connected to other devices (e.g., nodes or otherrouting devices) at initialization. Each port of a routing device maysense a characteristic of its communications link (e.g., voltage on areceive link) or may transmit a request and receive (or not receive) aresponse via its communications link to identify whether a device isconnected. The network manager may poll each routing device for anindication of which ports of the routing device are connected to otherdevices. The network manager can then send a query message to eachconnected-to port to identify the port to which it is connected.

[0021] In one embodiment, the network manager establishes paths throughthe network of routing devices by configuring the ports of the routingdevices along the path. The network manager may identify a path from asource node to a destination node using conventional path identificationtechniques. For example, the network manager may use a shortest pathalgorithm to identify the path with the smallest number ofcommunications links or may use a congestion-based algorithm thatfactors in actual or anticipated network traffic to identify the path.The network manager then identifies a virtual address (i.e., adestination virtual address) for the identified path. The virtualaddress is sent by the source node along with the data to be transmittedto the destination node. The data and virtual address may be stored in aframe (e.g., Fibre Channel or InfiniBand) that has a header and apayload. The header may contain the virtual address and the payload maycontain the data. The network manager then configures each source-sideport of each routing device along the path to forward frames sent to theidentified virtual address to the destination-side port of the routingdevice that is connected to the next communications link in the path.The configuration information may be stored in a label table (describedbelow) for the port that maps virtual addresses to destination-sideports. When a source-side port receives a frame with the identifiedvirtual address, it then forwards the frame through the destination-sideport in accordance with the configuration information.

[0022] In one embodiment, the network manager identifies a virtualaddress that is not currently in use by any source-side port along thepath. Thus, when a source-side port receives a frame addressed with theidentified virtual address, there is no ambiguity as which port of therouting device is the destination-side port. It is possible, however,that paths from two different source nodes to the same destination nodemay have a common sub-path. For example, the path from one source nodemay be through communications links A, X, Y, and Z, and the path fromthe other source node may be through communications links B, X, Y, andZ. In such a case, the network manager may use the same virtual addressfor both paths and share the terminal portion of the already-configuredpaths.

[0023] In one embodiment, the network manager may also establish a pathbetween the destination node and the source node. The network managermay identify a new path or may use the same path that was identifiedbetween the source node and the destination node (but in the oppositedirection). The network manager then identifies a virtual address (i.e.,source virtual address) and configures the ports along the path in amanner that is analogous to the configuration of the path from thesource node to the destination node. Whenever a source node sends aframe, it may include the source virtual address in the frame. When thedestination node receives the frame, it can respond to the source nodeby sending a frame addressed to the source virtual address.

[0024] In one embodiment, the network manager may need to identify andconfigure a new path between a source node and a destination node. Forexample, the network manager may determine that, because of congestion,the required quality of service cannot be provided along the existingpath or may detect a failure along the existing path. The networkmanager may be able to use the same virtual address to configure the newpath. If the network manager uses each virtual address only once, thenthe network manager can use the same virtual address for the new path.If, however, the same virtual address is used to identify differentpaths, then it may be possible that the configuration of the new pathmay conflict with the configuration of another path that uses the samevirtual address. When the same virtual address can be used, then thenetwork manager can change the path in a manner that is transparent tothe source node. In particular, the network manager need not notify thesource node of the change in the path. Also, if multiple destinationnodes provide the same functionality, then the network manager mayimplement node load balancing by dynamically changing a path so thatdata will be sent to a different destination node. The use of thesevirtual addresses allows the changes to be made without changing thesource and destination virtual addresses of the path.

[0025] In one embodiment, the network manager may reserve one or morevirtual addresses for sending frames from a device (e.g., routing deviceor node) to the network manager. For example, such a frame may include aregistration request from a source node. When the network manager isdistributed, a routing device may detect when it has received a framewith a reserved virtual address and may forward the frame directly tothe connected manager device for processing by the network manager. Toprovide flexibility, a frame directed to the network manager may includea combination of a reserved virtual address and another virtual address.When a routing device detects such a frame, it may determine whether itis configured to forward frames directed to the other virtual addressusing in-band communications. If so, the routing device forwards theframe through the destination-side port identified by the other virtualaddress. If the routing device is not configured for the other virtualaddress, then the routing device sends the frame to the network managervia out-of-band communications. For example, the routing device may sendthe frame to its directly connected manager device. In this way, thenetwork manager can configure the network so that certain frames areforwarded to certain manager devices that provide certain functions orservices of the network manager.

[0026] In one embodiment, a routing device is an interconnect fabricmodule (“IFM”) with high-speed switching capabilities. An interconnectfabric module can be dynamically configured to interconnect itscommunications ports so that data can be transmitted through theinterconnected ports. Multiple interconnect fabric modules can beconnected to form an interconnect fabric through which nodes (e.g.,computer systems) can be interconnected. In one embodiment, data istransmitted through the interconnect fabric as frames such as thosedefined by the Fibre Channel standard. Fibre Channel is defined in ANSIT11 FC-PH, FC-PH-2, FC-PH-3, FC-PI, and FC-FS industry standarddocuments which are hereby incorporated by reference. One skilled in theart will appreciate, however, that the described techniques can be usedwith communications standards other than Fibre Channel. In particular,the described techniques can be used with the InfiniBand standard, whichis described in the InfiniBand Architecture Specification, Vols. 1-2,Release 1.0, Oct. 24, 2000, which is hereby incorporated by reference.The interconnect fabric module may allow the creation of an interconnectfabric that is especially well suited for interconnecting devicesutilizing multiple information types such as might be required by thedevices of an enterprise data network (“EDN”).

[0027] In one embodiment, a virtual address may be part of a “virtualidentifier” (e.g., source or destination identifier) that includes adomain address. A destination identifier of a frame may be set to avirtual identifier. The destination identifiers of the frames receivedby the interconnect fabric modules are used to forward the frame. Eachinterconnect fabric module is assigned a domain address. Theinterconnect fabric modules that are assigned the same domain addressare in the same domain. The interconnect fabric modules use of thedomain addresses to forward frames between domains. The network managermay configure the interconnect fabric modules with inter-domain paths.When an interconnect fabric module receives a frame with a destinationdomain address that matches its domain address, then the frame hasarrived at its destination domain. The interconnect fabric module thenforwards the frame in accordance with the destination virtual addresssince it has arrived at its destination domain. If, however, the domainaddresses do not match, then the frame has not arrived at itsdestination domain. The interconnect fabric module forwards the frameusing an inter-domain path. Each port of an interconnect fabric modulemay have a domain address table (configured by the network manager) thatmaps the domain addresses to the destination port through which frameswith that domain address are to be forwarded. Thus, an interconnectfabric module may selectively use virtual addresses and domain addresseswhen forwarding frames.

[0028] In one embodiment, an interconnect fabric module uses acrosspoint switch to switch connect its source and destination ports.When the crosspoint switch has more switch ports than ports of theinterconnect fabric module, the extra switch port can be used foradministrative functions of the network manager. When an interconnectfabric module receives a frame directed to a virtual address reservedfor administrative services of the network manager, the interconnectfabric module connects the source port to the extra switch port which isconnected to a manager device. When the frame is transmitted from thesource port, the network manager at the manager device receives theframe and processes it in accordance with its administrative functions.In this way, administrative frames can be directly forwarded to thenetwork manager when they are first received by an interconnect fabricmodule from a node.

[0029] In some embodiments, one or more virtual identifier (“VI”)Network Interface Controller (“NIC”) facilities on each node (e.g., oneVI NIC for each network interface) facilitate the use of virtualidentifiers in communicating data. When a VI NIC on a node receives anindication that a data communication to one or more remote nodes is tooccur, such as from an application executing on the node, the VI NICwill identify an appropriate transmittal virtual identifier that can beused to route the data communication through the network to theappropriate remote destination nodes without being assigned to ordirectly associated with those destination nodes. Such datacommunications can include both transitory connectionless transmittalsof data (e.g., unidirectional transmittals from a source to adestination) and non-transitory connections that allow multiple distincttransmittals of data (e.g., a persistent dedicated connection thatallows a connection-initiating source and a connection destination totransmit data back and forth).

[0030] The VI NIC can identify an appropriate transmittal virtualidentifier for routing a data communication in various ways. In someembodiments, the VI NIC will register some or all outgoing datacommunications with a network manager for the network, and will receivean appropriate transmittal virtual identifier to be used for thatcommunication from the network manager. If an indicated datacommunication corresponds to a previously registered data communication(e.g., to an existing connection or to a previous communication to thesame destination and in the same transmission manner), however, the VINIC could instead in some embodiments use the previously receivedtransmittal virtual identifier for that data communication rather thanperform an additional registration for the indicated data communication.The manners in which a data communication can be transmitted vary withthe transmission characteristics that are supported by a network, andcan include factors such as a particular Class Of Service (“COS”) ortransmission priority.

[0031] In some embodiments, when a data communication indicated by asource can result in bi-directional communication (e.g., a response fromone or more of the destinations), the VI NIC also identifies a responsevirtual identifier that can be used for routing data from one or more ofthe destinations back to the source. If the VI NIC registers the datacommunication with a network manager, this response virtual identifiermay be received from the network manager. After identifying thisresponse virtual identifier, the VI NIC associates it with informationindicating how to process received data communications that are routedusing the response virtual identifier. In some embodiments, suchreceived data communications are processed by forwarding the datacommunications to one or more resources associated with the destinationnode, such as an executing application program, a file on storage, or adevice that is part of the node. For example, if a source application ona source node initiates a bi-directional communication, a VI NIC for thesource node may associate the response virtual identifier with thatsource application so that received responses can be forwarded to thatsource application (which then becomes the destination application forthose received communications).

[0032] For illustrative purposes, some embodiments are described belowin which the VI NIC is used as part of a Fibre Channel or InfiniBandnetwork and/or as part of an EDN architecture. However, those skilled inthe art will appreciate that the techniques of the invention can be usedin a wide variety of other situations and with other types of networks,and that the invention is not limited to use in Fibre Channel orInfiniBand networks or with EDN architectures.

[0033]FIG. 1 is a network diagram illustrating various nodes of anexample Fibre Channel fabric-based interconnect network that areinter-communicating using virtual identifiers. In this exampleembodiment, multiple interconnect fabric modules (“IFMs”) 110 withhigh-speed switching capabilities are used as intermediate routingdevices to form an interconnect fabric, and multiple nodes 105, anetwork manager 115 and a Multi-Protocol Edge Switch (“MPEX”) 120 areconnected to the fabric. Each of the nodes has at least one VI NIC thatuses virtual identifiers when communicating and receiving data. The MPEXis used to connect the Fibre Channel or InfiniBand network to anexternal network, such as an Ethernet-based network, and similarlyincludes at least one VI NIC. Data is transmitted through theinterconnect fabric using frames such as those defined by the FibreChannel or InfiniBand standards.

[0034] Topology Discovery

[0035] As described above, the network manager may dynamically discoverthe topology of the network using various different techniques. In theembodiment described below, each interconnect fabric module identifieswhich of its ports are connected to other devices. The network manageruses this information to send a message through each port that isconnected to another device to identify the connected-to device. FIG. 2is a flow diagram illustrating the discovery processing of a componentof the interconnect fabric module in one embodiment. Each port of aninterconnect fabric module identifies whether it is connected to a portof another device, such as another switch or a node. The interconnectfabric module then provides to the network manager an indication ofwhich of its ports are connected to other ports to assist in thediscovery process. In blocks 201-204, the component determines whethereach port is currently connected to another port. In block 201, thecomponent selects the next port. In decision block 202, if all the portshave already been selected, then the component completes, else thecomponent continues at block 203. In decision block 203, the componentdetermines whether the selected port is connected to another port. Thisdetermination may be made based on various voltage levels of thecommunications links. If there is a connection, then the componentcontinues at block 204, else the component loops to block 201 to selectthe next port of the interconnect fabric module. In block 204, thecomponent notes the selected port as connected to another port and loopsto block 201 to select the next port of the interconnect fabric module.

[0036]FIG. 3 is a flow diagram illustrating the discovery processing ofthe network manager in one embodiment. The network manager firstretrieves an indication of which ports of the interconnect fabricmodules are connected to other devices. The network manager then sends aquery message through each of the indicated ports to the connected-toport. When the connected-to port receives the query message, it respondswith an identification of its interconnect fabric module and its portnumber. In this way, the network manager can discover the topology ofthe interconnect fabric. In blocks 301-303, the network managerretrieves the indications of which ports of the interconnect fabricmodules are connected to other ports. In block 301, the network managerselects the next interconnect fabric module that has not yet beenselected. In decision block 302, if all the interconnect fabric moduleshave already been selected, then the network manager continues at block304, else the network manager continues at block 303. In block 303, thenetwork manager retrieves an indication of which ports of the selectedinterconnect fabric module are connected to other ports. The networkmanager may send the message using either in-band our out-of-bandcommunications. The network manager then loops to block 301 to selectthe next interconnect fabric module. In blocks 304-310, the networkmanager determines the identity of each of the connected-to ports. Inblock 304, the network manager selects the next interconnect fabricmodule. In decision block 304, if all the interconnect fabric moduleshave already been selected, then the network manager completes itsdiscovery process, else the network manager continues at block 306. Inblocks 306-310, the network manager loops sending a query messagethrough each port of the selected interconnect fabric module that isconnected to another port. In block 306, the network manager selects thenext port of the selected interconnect fabric module that is connectedto another port. In decision block 307, if all such ports are alreadyselected, then the network manager loops to block 304 to select the nextinterconnect fabric module, else the network manager continues at block308. In block 308, the network manager sends a query message through theselected port of the selected interconnect fabric module. In block 309,the network manager receives the identification of the connected-to portof the selected port of the selected interconnect fabric module. Theidentification may include an indication of the interconnect fabricmodule and the port number of the connected-to port. In block 310, thenetwork manager stores a mapping between the selected port of theselected interconnect fabric module and the connected-to port of theconnected-to interconnect fabric module. These mappings define thetopology of the network. The network manager then loops to block 306 toselect the next port of the selected interconnect fabric module that isconnected to another device.

[0037] The processing of the discovery of the network manager asdescribed above assumes that the network manager initially is aware ofall interconnect fabric modules of the interconnect fabric. One skilledin the art will appreciate that the network manager may become of awareadditional interconnect fabric modules during the discovery process. Forexample, if the network manager is centralized, then it may initiallysend a query message through its port that is connected to theinterconnect fabric. The receiving port responds with the identity andinterconnect fabric module and its port number. The network manager canthen requested that identified the interconnect fabric module to providea indication of which of its ports are connected to other ports. Thenetwork manager can then send a query message through each of theindicated ports to the connected-to ports. The connected-to ports thenrespond with the identification of the connected-to interconnect fabricmodule and connected-to port. This process can be repeated transitivelyby the network manager to identify all interconnect fabric modules thatcomprise the interconnect fabric.

[0038] Establishing a Path

[0039]FIG. 4 is a flow diagram illustrating the process of establishinga path by the network manager in one embodiment. A path is typicallyestablished when a node registers with the network manager. An establishpath component of the network manager may receive an indication of asource node and a destination and then identify paths of ports ofinterconnect fabric modules from the source node to the destination nodeand from the destination node to the source node. The component thenidentifies virtual addresses for the paths and initializes the labeltables of the ports of the interconnect fabric modules along theidentified paths. A label table of a port contains mappings from virtualaddresses to destination-side ports through which a frame sent to thatvirtual address is to be forwarded. In block 401, the componentidentifies the paths. In one embodiment, the path from the source nodeto the destination node and the path from the destination node to thesource node use the same ports of the same interconnect fabric modules.That is, the paths use the same communications links. Alternatively, thepath in one direction may be different from the path in the otherdirection. One skilled in the art will appreciate that variouswell-known techniques for identifying paths can be used. In block 402,the component invokes an identify virtual address component passing anindication of the path and an indication that the virtual address to beused by the source node when sending a communications to the destinationnode (e.g., the destination virtual address). The invoked component mayselect a virtual address that is not currently in use by any of thesource-side ports of the path. A source-side port of the path is a portthat receives data sent by a source node, and a destination-side port ofthe path is a port through which data is transmitted on its way to thedestination node. In block 403, the component invokes in identifyvirtual address component passing an indication of the path and that thevirtual address is to be used by the destination node (e.g., the sourcevirtual address). In block 404, the component invokes a component toinitialize the label tables of the source-side ports of the path withthe destination virtual address. The invoked component transmitsinstructions to the each source-side port of the path indicating thatthe port is to update its label table to map the source virtual addressto a destination-side port of the interconnect fabric module. In block405, the component invokes a component to initialize the label tables ofthe destination-side ports of the path with the source virtual address.The component then completes.

[0040]FIG. 5 is a flow diagram illustrating the processing of anidentify virtual address component of the network manager in oneembodiment. In this embodiment, the identify virtual address componentis provided an indication and a path along with an indication of whethera virtual address for the source node or the destination node is to beidentified. The component may check every port along the path toidentify a virtual address that is not currently used by a port alongthe path. Alternatively, the component may identify virtual addressesbased on a sequential ordering. That is, the component may keep track ofthe last identified virtual address and increment that virtual addressto identify the next virtual address. In this way, each virtual addressis unique. In blocks 501-505, the component loops selecting the nextvirtual address and determining whether it is available. The virtualaddress may not be available to a port along the path when that portalready uses that virtual address. In blocks 501, the component selectsto the next virtual address. In decision block 502, if all the virtualaddresses have already been selected, then the component indicates thata virtual address could not be identified, else the component continuesat block 503. In blocks 503-505, the component loops selecting each portalong the path and determining whether that port already uses theselected virtual address. In block 503, the component selects the nextinterconnect fabric module and port of the path. In decision block 504,if all the interconnect fabric modules and ports of the path havealready been selected, then the component uses the selected virtualaddress as the identified virtual address and then completes, else thecomponent continues at block 505. In decision block 505, if the selectedvirtual address is available at the selected interconnect fabric moduleand selected port, then the component loops to block 503 to select thenext port along the path, else the component loops to block 501 toselect the next virtual address.

[0041]FIG. 6 is a flow diagram illustrating the processing of aninitialize label table component of the network manager in oneembodiment. The initialize label table component sends a command to eachport along the path indicating to add a mapping from the identifiedvirtual address to the other port of that interconnect fabric modulealong the path. The component is passed in indication of the path, thevirtual address, and an indication of whether the virtual address is asource virtual address or a destination virtual address. In block 601,the component selects the next interconnect fabric module and port inthe path based on whether the source or destination virtual address hasbeen passed. In decision block 602, if all the interconnect fabricmodules along the path have already been selected, then the componentcompletes, else the component continues at block 603. In block 603, thecomponent sends a message to be selected port of the interconnect fabricmodule indicating to add to its label table a mapping from the virtualaddress to the other port of the path. The component then loops to block601 to select the next interconnect fabric module and port in the path.

[0042] Reserved Addressing

[0043] In one embodiment, the crosspoint switch of an IFM may have moreoutputs than the number of ports of the IFM. For example, a crosspointswitch may have 34 inputs and outputs, but the IFM may have only 32ports. The IFM may use these additional ports of the crosspoint switchto route upper layer protocol frames, such as frames directed into aname server or other administrative services. In one embodiment, theadditional output ports of the crosspoint switch may be connected to amanager device for the IFM. An interconnect fabric module may have alist of “reserved” addresses that designate an upper layer protocolport. When an IFM determines that an address of its frame matches one ofthe reserved addresses, it enables the routing of that frame to an upperlayer protocol port. The routing to upper layer protocol ports may usethe same arbitration mechanism as used for routing to non-upper layerprotocol ports as described in the Patent Application entitled“Interconnect Fabric Module.” Alternatively, when the crosspoint switchdoes not have extra output for an upper layer protocol port, an outputcan be selectively switched between a communications port and an upperlayer protocol port depending on whether the address of the destinationidentifier is reserved.

[0044]FIG. 7 is a block diagram illustrating a distributed networkmanager in one embodiment. In this embodiment, the network manager maybe implemented on a series of manager devices connected directly to theinterconnect fabric modules. The distributed network manager maycommunicate with each other using in-band communication of theinterconnect fabric or using out-of-band communication that isindependent of the interconnect fabric. The crosspoint switch of aninterconnect fabric module may have reserved ports for the distributednetwork manager. When an interconnect fabric module receives data thatdesignates one of the reserved ports, then the interconnect fabricmodule forwards the data to the distributed network manager through thereserved port.

[0045]FIG. 8 is a flow diagram illustrating the processing of acomponent of an interconnect fabric module that processes reservedaddresses in one embodiment. This component forwards the frame to thenetwork manager via either in-band or out-of-band communications. Withthe use of in-band communications the frame can be routed to theappropriate interconnect fabric module, which can then send the frame tothe network manager using the out-of-band communications. In block 801,if the virtual address of the received frame is a reserved address, thenthe component continues at block 802, else the component completes. Indecision block 802, if the virtual address parameter within the frame isin the label table, then the frame is to be forwarded using in-bandcommunications and the component continues at block 804, else the frameis to be forwarded directly to the network manager at the IFM's managerdevice using out-of-band communications and the component continues atblock 803. In block 803, the component forwards frame to theadministrative port and then completes. In block 804, the componentforwards the frame based on the port map of the label table and thencompletes.

[0046] One skilled in the art will appreciate that, although variousembodiments of the technology have been described, various modificationsmay be made without deviating from the spirit and scope of theinvention. Accordingly, the invention is not limited except as by theappended claims.

1. A method for identifying topology of a network, the network includinga plurality of switches, each switch having ports, each port of a switcheither being connected to another port or not connected to another port,the method comprising: under control of each switch, determining whethereach port of the switch is connected to a connected-to port; and undercontrol of a network manager, for each of the switches, retrieving anindication of which of the ports of the switch are connected to aconnected-to port; and for each port that is connect to a connected-toport, sending a query message through that port to the connected-toport; and receiving a response from the connected-to port identifyingthe connected-to device and connected-to port wherein mappings from eachswitch and port to its connect-to device and connected-to port indicatesthe topology of the network.
 2. The method of claim 1 wherein processingof the network manager is distributed to the switches.
 3. The method ofclaim 1 wherein the query message is sent via out-of-bandcommunications.
 4. The method of claim 1 wherein the sending of theconnect-to query message is sent via in-band communications of thenetwork.
 5. The method of claim 4 wherein the network manager identifiesswitches of the network via the received responses.
 6. The method ofclaim 5 wherein when a switch is identified, the network managerperforms the retrieving of the indications of which of the ports of theswitch are connected to a connected-to port.
 7. The method of claim 1wherein the connected-to device is a node.
 8. The method of claim 1wherein the connected-to device is a switch.
 9. A method for identifyingtopology of a network, the network including a plurality of routingdevices, each routing device having ports, the method comprising:retrieving an indication of which of the ports of the routing devicesare connected to another port; for each port that is connect to anotherport, sending a query message through that port to the other port; andreceiving a response from the other port identifying the other deviceand the other port.
 10. The method of claim 9 including generating amapping from each routing device and port to device and port to which itis connected to indicate the topology of the network.
 11. The method ofclaim 9 wherein a routing device is a switch.
 12. The method of claim 9wherein a routing device is an interconnect fabric module.
 13. Themethod of claim 9 wherein the routing devices use virtual addresses toroute frames.
 14. The method of claim 9 wherein the identification ofthe topology is performed by a network manager.
 15. The method of claim14 wherein the network manager is distributed to the routing devices.16. The method of claim 9 wherein the query message is sent viaout-of-band communications.
 17. The method of claim 9 wherein the querymessage is sent via in-band communications.
 18. The method of claim 9wherein the routing devices of the network are identified via thereceived responses.
 19. The method of claim 18 wherein when a routingdevice is identified, retrieving an indication of which of the ports ofthe routing device are connected to another port.
 20. The method ofclaim 9 wherein the retrieving of an indication of which of the ports ofthe routing devices are connected to another port includes sending arequest to the routing device.
 21. The method of claim 9 wherein theretrieving of an indication of which ports of the routing devices areconnected to another port includes receiving a message from the routingdevice.
 22. The method of claim 9 wherein each routing device determineswhich of its ports are connected to another port and the retrieving ofan indication of which of the ports of the routing devices are connectedto another port includes transmitting the determined information to anetwork manager.
 23. A network manager for identifying topology of anetwork, the network including a plurality of routing devices, eachrouting device having ports, comprising: a component that retrievesindications of which of the ports of the routing devices are connectedto another port; and a component that sends a query message through eachport that is indicated as connected to another port to the other portand that receives a response from the other port identifying the otherdevice and the other port.
 24. The network manager of claim 23 includinga component that generates a mapping from each routing device and portto the device and port to which it is connected to indicate the topologyof the network.
 25. The network manager of claim 23 wherein a routingdevice is a switch.
 26. The network manager of claim 23 wherein arouting device is an interconnect fabric module.
 27. The network managerof claim 23 wherein the routing devices use virtual addresses to routemessages.
 28. The network manager of claim 27 including a component thatconfigures each routing device with routing data for virtual addresses.29. The network manager of claim 28 wherein each frame of dataidentifies a destination virtual address.
 30. The network manager ofclaim 23 wherein the query message is sent via out-of-bandcommunications.
 31. The network manager of claim 23 wherein the querymessage is sent via in-band communications.
 32. The network manager ofclaim 23 wherein the routing devices of the network are identified viathe received responses.
 33. The network manager of claim 32 wherein thecomponent that retrieves an indication of which of the ports of therouting device are connected to a another port retrieves the indicationwhen a routing device is identified.
 34. The network manager of claim 33wherein the component that retrieves an indication sends a request to arouting device.
 35. The network manager of claim 23 wherein thecomponent that retrieves an indication of which ports of the routingdevices are connected to another port includes receiving a message fromthe routing device.
 36. The network manager of claim 23 wherein eachrouting device determines which of its ports are connected to otherports and the retrieving of an indication of which of the ports of therouting devices are connected to another port includes receiving thedeterminations from the routing devices.
 37. A network manager foridentifying topology of a network, the network including a plurality ofrouting devices, each routing device having ports, comprising: means forretrieving indications of which of the ports of the routing devices areconnected to another port; and means for sending a query message througheach port that is indicated as connected to another port to the otherport and that receives a response from the other port identifying theother port.
 38. The network manager of claim 37 including a componentthat generates a mapping from each port to its connected-to port toindicate the topology of the network.
 39. The network manager of claim37 wherein a routing device is a switch.
 40. The network manager ofclaim 37 wherein a routing device is an interconnect fabric module. 41.The network manager of claim 37 wherein the routing devices use virtualaddresses to route messages.